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Abstract 

The utility of the Icing Contamination Envelope Protection (ICEPro) system for mitigating a 
potentially hazardous icing condition was evaluated by 29 pilots using the NASA Ice Contamination 
Effects Flight Training Device (1CEFTD). ICEPro provides real time envelope protection cues and 
alerting messages on pilot displays. The pilots participating in this test were divided into two groups; a 
control group using baseline displays without ICEPro, and an experimental group using ICEPro driven 
display cueing. Each group flew identical precision approach and missed approach procedures with a 
simulated failure case icing condition. Pilot performance, workload, and survey questionnaires were 
collected for both groups of pilots. Results showed that real time assessment cues were effective in 
reducing the number of potentially hazardous upset events and in lessening exposure to loss of control 
following an incipient upset condition. Pilot workload with the added ICEPro displays was not 
measurably affected, but pilot opinion surveys showed that real time cueing greatly improved their 
situation awareness of a hazardous aircraft state. 


Nomenclature 


a 

Alternate Hypothesis 

AOA 

Angle of Attack 

D-ICES 

Dynamic Inversion Control Evaluation System 

1CEFTD 

Ice Contamination Effects Flight Training Device 

ICEPro 

Icing Contamination Envelope Protection System 

ISP 

Icing Severity Parameter 

H 0 

Null Hypothesis 

PCE 

Pilot Coupling Event 

RTP1D 

Real Time Parameter Identification 
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1.0 Introduction 

The University of Tennessee Space Institute (UTSI) in partnership with Bihrle Applied Research 
(BAR) completed a 3 -year cooperative research effort with NASA under NASA Research Announcement 
(NRA) NNH06ZEA001N, Appendix B, of the Aviation Safety Program. The objectives of the NRA were 
defined under the Integrated Vehicle Health Management (IVHM) Project, topic IVHM 3.1, 
Environmental Hazards, which are caused by the “Effects of Icing on Aircraft State”. This effort 
culminated in the development of a real time vehicle state assessment system, which was described by 
Gingras (Ref. 1) and is referred to as the Icing Contamination Envelope Protection system (ICEPro). The 
algorithms in ICEPro were initially developed and tested in a desktop simulation environment, and then 
integrated and further developed in the NASA Ice Contamination Effects Flight Training Device 
(ICEFTD). The two flight models used in simulation to develop ICEPro fortesting and evaluation were 
based upon the characteristics of the NASA Twin Otter icing research aircraft in a no-ice baseline 
condition, and those representing a failure of the ice protection system (IPS). The failure case was defined 
by the ice accretions that would form during a 22.5 min FAR 25 Appendix C icing encounter as 
calculated by LEWICE version 2.0. Most of the data for the simulator came from testing a 6.5 percent 
scale Twin Otter model in the Wichita State University 7X10 Low Speed wind tunnel and BAR’s Large 
Amplitude Multi-Purpose (LAMP) rotary balance wind tunnel using scaled ice shapes (Ref. 2). Flight data 
from aircraft testing with failure case ice shapes was also gathered to refine the sub-scale model data so it 
would match full scale flight test results (Ref. 3). 

After development and testing in a desktop environment, ICEPro was integrated with the NASA 
ICEFTD (Refs. 4 and 5). Researchers from BAR and UTSI completed final development and integration 
testing to ensure readiness for the planned pilot evaluations. A final check of test readiness was then 
performed by having two experienced NASA test pilots complete the evaluation profile, and all required 
pre-test preparation and training. This included a review of: the test plan, NASA web based icing training, 
standard operating procedures (SOP) for the Twin Otter, the NASA Task Load Index or TLX workload 
method (Ref. 6), a tutorial on flying with ICEPro displays (only for the pilot using the experimental 
displays), and post-test survey questionnaire. One NASA pilot was selected to represent the control group 
and flew with baseline displays. The other was selected to represent the experimental group and flew with 
the ICEPro modified displays. Before the test began, each pilot was briefed on the entire test process. The 
test protocol at the time called for minimal training in aircraft handling with hazardous ice formations, 
somewhat similar to that which a typical pilot would receive in an aircraft transition course. However, 
once testing began, it became apparent that the flight task was so difficult that without practice, there was 
a high probability of a random “crash” regardless of the displays used. It was apparent that a different 
approach would have to be taken. After assessing the pilot’s comments from the debriefings, it was 
therefore decided to modify the test protocol and train the pilots in both groups to proficiency for 
executing the hazardous approach task using their respective flight displays. Utility benefits could then be 
assessed by comparing pilot performance, workload, and opinions between control and experimental 
groups who were flying the same task with the same training and preparation, but with different flight 
displays. This adjustment in the test plan was an essential change, and would not have been discovered 
without this pre-test. The NASA pilots also completed the NASA TLX forms and answered all the survey 
questions during their respective evaluations. A few small changes in the survey questions were also 
required to eliminate ambiguity problems. After these adjustments were made to the test plan, pilot-in- 
the- loop (PIL) evaluations of ICEPro were scheduled at the Embry Riddle Aeronautical University 
(ERAU), Daytona Beach, Florida. 

2.0 Background 

Ice formations on aircraft can negatively affect performance, stability and control characteristics, and 
handling qualities. Accidents that have occurred due to these issues have been documented by the 
National Transportation Safety Board (NTSB) and the Federal Aviation Administration (FAA) over many 
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years. In a recent study (Ref. 7) of aircraft icing accidents and incidents reported between 1978 and 2002, 
loss of control events were usually preceded by stall and more likely to occur during approach and 
landing. This is the busiest phase of flight where historically over 50 percent of all accidents occur, and 
where human error is responsible for approximately 75 percent of those accidents (Ref. 8). Ice 
contamination, which accretes on a critical airframe surface such as the wings and tail, due to an Ice 
Protection System (IPS) failure, encountering an exceedance condition, or operator error can have a 
detrimental effect on aircraft performance, stability and control, and handling qualities. This situation is 
further exacerbated by the required changes in airspeed and configuration as the pilot slows and 
configures during the descent, approach, and landing phases. A lateral upset resulted in the fatal crash of 
American Eagle 4184 after encountering what appeared to be an exceedance icing condition. The NTSB 
listed the probable cause of the accident as . .the loss of control, attributed to a sudden and unexpected 
aileron hinge moment reversal that occurred after a ridge of ice accreted beyond the deice boot and 
ultimately lead to an upset condition” (Ref. 9). The pilots of this aircraft had no forewarning of the 
impending upset condition and furthermore, were flying on auto pilot at the time of the upset, which 
masked the rapid build-up in lateral control force as the flaps were raised while descending in the holding 
pattern. As the upset occurred, the autopilot disengaged and the aircraft rolled sharply to the right. The 
pilots then attempted to recover the aircraft, but in so doing entered a secondary stall upset, which caused 
the aircraft to roll inverted and enter a steep dive and crash. In its Tailplane Icing Program (TIP) NASA 
employed various artificial ice shapes, which were attached to the horizontal tail plane of a research 
aircraft (Ref. 10). Numerous test maneuvers were conducted where wing flap settings, engine power, and 
flight speed were varied. The results showed a considerable decrease in longitudinal stability and elevator 
control effectiveness as a function of increasing wing flap angle, power setting, and airspeed. In the 
extreme, a tail stall upset would occur, which resulted in an uncontrollable nose down pitching motion. 
Airframe ice contamination also increases drag, reducing climb performance and is most evident in 
propeller driven aircraft. Although not a handling issue per se, it can greatly reduce engine out ceiling 
(Ref. 11), which in multi-engine aircraft and could ultimately result in controlled flight into terrain or 
CFIT. This is especially a concern if an engine should fail during a missed approach procedure. In cases 
such as these, wing flap settings must be reduced and speeds increased to ensure a positive rate of climb 
while avoiding a stall upset. Unfortunately, pilots cannot always determine when the state of the aircraft is 
adversely degrading, nor are they always able to predict how a configuration change such as raising or 
lowering the wing flaps, would affect aircraft stability and control. The ICEPro system was therefore 
developed with the theory that a knowledge -based system, which compares predicted aircraft stability and 
control characteristics with those existing in real time, could effectively provide envelope protection cues 
and alerting messages to help the pilot keep the aircraft in a safe condition of flight. The knowledge base 
consists of the baseline or clean aircraft model, and an ice contamination model with known safe flight 
envelope limits as demonstrated by either wind tunnel or flight test. Angle of attack (AO A) cues, which 
are driven by ICEPro, show the pilot safe limits that will protect against either a wing stall or tail stall 
upset event. Messages, on the other hand, alert the pilot of loss in stability and control as a function of 
flight configuration and condition, and of configuration limits as a function of icing condition and wing 
flap setting. 

3.0 ICEPro System Description 

3.1 System Design and Architecture 

During the initial system design stage of the program, the research team decided upon a real time 
state assessment system architecture, which would incoiporate two state estimation methodologies; a 
Dynamic Inversion Control Evaluation System (D-ICES), which assesses control position from measured 
aircraft response, and a Real Time Parameter Identification (RTPID) (Refs. 12, 13, and 14) system, which 
estimates stability and control parameters from manual or automatic flight control system inputs and the 
resulting aircraft response. ICEPro is controlled by executive logic, which 
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uses D-ICES to compare existing control deflections with those predicted from a knowledge base of the 
“clean” or un-iced aircraft. As shown in Figure 1, the system architecture consists of three modes of 
operation, defined as Monitor, Identify or ID, and Report. In the “MONITOR” mode, measured control 
surface deflections are compared to predicted control surface deflections from the baseline aircraft model 
in the D-ICES algorithm. If the deflections of the real-time aircraft differ from those of the baseline model 
by a defined threshold value, ICEPro switches to the “ID” mode. The system logic executes this mode 
change by using a normalized statistical metric called a Theil (Ref. 15) coefficient, which is calculated 
from control deflection samples taken in a moving time window. This metric indicates the level of 
agreement between the actual control deflection and the D-ICES prediction. In this mode, ICEPro 
diagnoses the degraded vehicle condition by executing a series of optimally-designed control excitations 
(Ref. 1 6) for 1 0 sec to provide information for the RTPID algorithm to estimate stability and control 
derivatives from measured control surface deflections and vehicle response. An “Icing Severity 
Parameter” or ISP is calculated based upon the time averaged differences between the a priori clean and 
iced values. When the ISP exceeds a threshold amount, ’’Report” mode is entered and flyable cues and 
alerting messages relating to the severity of the condition are provided on pilot displays. These displays 
provide a prognosis of the condition and envelope protection limits. This process is iterative, and 
continues throughout the duration of the flight. If at any time the ISP drops below a predetermined 
threshold, cueing and messages will disappear and the system returns to the monitor mode. The system 
also consists of latching and de-latching timers, not shown in the figure, which prevent nuisance cueing 
and message toggling. It is beyond the scope of this paper to provide specific system design details. These 
details however, will be discussed in a companion paper by David Gingras, which will be presented at the 
AIAA Modeling and Simulation Conference, Toronto, Ontario, Canada, August 2 to 5, 2010. 


3.2 Functionality Description and Flight Displays 

The fundamental function of ICEPro is to perform real time state assessment and provide cues and 
messages to the flight deck that assist the pilot in keeping the aircraft in a safe operating envelope at all 
times. Figure 2 shows how those cues and messages are implemented on flight displays. 

Airspeed carets provide high and low airspeed limits for the current aircraft configuration and 
estimated state due to icing. In general, the high speed limit is based upon longitudinal stability and 
control considerations, and the low speed limit is based on wing stall speed. The carets appear next to the 
airspeed tape and are marked with a snowflake symbol to indicate that they are estimated by ICEPro. 
Their position on the tape depends upon wing flap setting and the estimated aircraft state due to icing. 
Figure 2 shows a notional display of the high speed limit (rectangular wedge caret) corresponding to tail 
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Figure 2. — ICEPro flight displays. 


stall upset speed, and the low speed limit (triangular caret) corresponding to the stall speed with wing- 
flaps in the landing position. When the wing flaps are not in the landing position, the triangular stall speed 
caret is amber and provides stall warning corresponding to 1 07 percent of the stall speed. 

AOA brackets provide maneuvering limits in pitch for the pilot. These brackets are superimposed on the 
pitch ladder of the attitude indicator display. There is an upper bracket that indicates the predicted stall AOA 
for the current configuration and ice contamination state, and a lower bracket that indicates the minimum safe 
AOA to provide tail stall and/or negative-g protection. The brackets are nominally white when the aircraft is 
being flown at a safe mid range AOA between the brackets. When the AOA approaches either the upper or 
lower bracket, both brackets turn amber to indicate caution, and when the AOA is at or beyond the bracket 
levels, the brackets turn red to warn of an unsafe condition. In addition, whenever the brackets turn amber or 
red, they flash for several seconds to ensure that the pilot notices the change. The relationship between upper 
and lower bracket depend upon the AOA limits for the current wing flap configuration. 

ICEPro provides a flap position indication that advises the pilot as to the state of the vehicle at the 
current flap deflection. As flaps are deployed during icing conditions and the system detects degradations 
in stability and control and/or in the single-engine climb capability, the indicator will change to an amber 
color, cautioning further deployment of the flaps. As the condition worsens, the indicator turns red, 
warning the pilot to stop deployment and retract the flaps until the condition is cleared. 

Stability and control alerting messages are provided to inform the pilot of an adverse change in the 
stability and or control in the pitch, roll, or yaw axes. These messages on the Primary Flight Display 
(PFD) are repeated on the Flight Control Status display, which is shown to the right of the PFD on 
Figure 2. Thresholds for displaying the alerting messages are activated when pre-established reductions 
are determined from real time parameter estimates. An amber PITCH DGRD message for example, 
means that pitch stability or controllability has degraded by 50 percent or more than that of the nominal 
un-iced aircraft. The message turns red when degradation is 75 percent or greater than nominal. To 
enhance pilot awareness, the color of the primary or secondary control surface in the affected control axis 
is also changed to that of the message. This functionality prompts the pilot to return to a previous 
configuration and flight condition that provided safer flight characteristics. 
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A climb limit (CLIMB LIM) caution and warning message is a performance-based cue to provide the 
pilot with an indication of single-engine climb potential given the current state of the aircraft. An amber 
message indicates that should an engine fail the airplane climb rate for the current flap setting will be less 
than 100 ft per minute. If the message is red, this indicates that the airplane will be unable to climb or will 
descend in the event of an engine failure. 

Later in development, it was also decided to incorporate a “stick shaker” for stall warning, a STALL 
message, stick pusher and aural warning for a stall upset and a de-cluttering mode for the primary flight 
display when flight attitudes became excessive during extreme upset conditions. De-cluttering removes 
all but primary recovery cues such as AOA and airspeed limits. 

4.0 Test Methodology 

4.1 Experiment Design 

The purpose of the test and evaluation with pilots in the loop was to determine if ICEPro had utility 
for mitigating a potentially hazardous icing encounter. The independent parameters were the two displays 
(baseline, and ICEPro with envelope protection), and the dependent parameters were technical 
performance, workload, and pilot opinion. A non-parametric, independent measures experiment was 
therefore selected where differences in the dependent variables were analyzed using the Mann-Whitney 
“U” test (Ref. 17). A critical “U” was determined, based on a one-tailed test where a 95 percent 
confidence criteria or better was required to correctly reject the null hypothesis (H 0 ). H 0 assumed that 
there was no difference between the two groups for the tested variable, therefore the level of significance 
or alpha level required to reject the null hypothesis was chosen to be a = 0.05. The experiment was 
planned for an evaluation by 30 pilots with relatively similar flight experience. All were ERAU instructor 
pilots with a median flight experience of approximately 1300 flight hours. Each possessed commercial, 
multi-engine, and instrument ratings, but none had any specific experience in actual icing conditions or 
any prior training in aircraft handling and upset recovery due to a hazardous icing condition. The 30 pilots 
were randomly divided into a control group (baseline flight displays and no ICEPro system) and an 
experimental group (displays modified with ICEPro cues and messages). A scheduling problem resulted 
in a total of 29 pilot evaluations and unequal sample sizes between the control and experimental groups 
(n c = 14 and n e = 15). This disparity was accounted for by the MWU methodology as provided for in 
Reference 17. 

4.2 ICEFTD Simulator Description 

The aircraft model and sub-system components for the ICEFTD simulator were assembled in BAR’s 
commercial off the shelf (COTS) proprietary simulator development environment, D-Six (Bihrle Applied 
Research, Inc.). The simulator provides highly representative flight characteristics of the Twin Otter icing 
research aircraft. Although originally designed as a pilot training device, the ICEFTD’s ability to 
accurately represent aircraft icing characteristics made it a very usefiil research tool with several 
advantageous features. One is its capability for accurately representing flight characteristics beyond stall 
AOA, which permits pilots to experience post-stall gyrations and tail stall hard-over’s that occur during 
upset conditions. Another is the pitch control system, which can provide control force gradients and 
deflections that closely match those recorded from flight tests of the NASA Twin Otter during upset 
events. Lastly, an “aircraft emulator” was designed and integrated in order to represent the same sensor 
feedback and output response as that of the aircraft. Much of the functionality testing during development 
focused on refining the system logic to fine tune thresholds and minimize nuisance messaging. Figure 3 is 
an overview of the ICEFTD as modified for this test. The items in red font are those which support 
ICEPro operation. 
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Figure 3. — ICEFTD configuration for test and evaluation of ICEPro. 


The ICEFTD simulator also provided excellent three-screen graphics of a visual scene, which was an 
essential element of the final approach and missed approach task in this evaluation. A video recording 
camera system was also employed for each pilot evaluation run in the event a replay of the pilot’s actions 
needed to be observed. During testing, the evaluation pilot wore a headset and was completely enclosed in 
the simulator by a system of curtains (not shown in the figure). This isolated the pilot from any outside 
disturbances or distractions, and enhanced the fidelity of the simulation environment. A member of the test 
team operated the simulator and performed data acquisition and processing tasks from a remote console. A 
second member communicated with the pilot while acting as test conductor and air traffic control (ATC). 

4.3 Pre-Test Pilot Proficiency Training 

Before performing the evaluation tasks for data collection, each pilot was subjected to approximately 
1.5 hr of flight training. Pilots were trained for the evaluation task using the respective flight displays 
depending on whether they were in the control or experimental group. Training began by having each 
pilot execute the evaluation approach task profile as shown in Figure 4 in a no-ice or baseline 
configuration under instrument meteorological conditions (IMC). The IMC condition for training and 
testing was set at 400 ft and 1 nautical mile (NM) visibility. The white boxes in the figure describe the 
flight condition and events during the task and the green shaded boxes show the commands given by the 
test conductor, who acted as ATC at various stages during its execution. The missed approach procedure 
(MAP) with an engine failure was initiated by the test conductor at the approximate position as shown in 
red font. Pilots practiced the approach task according to the SOP, which specified speeds, configurations, 
and performance standards throughout the profile. This phase of training was important in order for the 
test team to determine that each evaluation pilot possessed the requisite instrument flight skills to 
satisfactorily perform the task. The next phase of training was conducted up and away under visual 
meteorological conditions (VMC) to familiarize pilots and develop their skill in dealing with aircraft 
characteristics due to the failure case icing condition. This training emphasized recognition of impending 
upset conditions and the effects of wing flap configuration and flight speed on aircraft handling and 
control characteristics. Pilots in the control group were taught how to most effectively use their baseline 
flight displays to control the aircraft and recognize and recover from upset conditions. Pilots in the 
experimental group received the same training emphasis, but were taught how to integrate their scan and 
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Figure 4. — Evaluation profile approach task flown by each pilot group during ICEPro testing and evaluation. 


control strategy with envelope protection cues to facilitate safe flight control and upset recoveries. During 
this training, evaluation pilots were also instructed to fly with both hands on the yoke. For some pilots 
this was a learning experience because nearly all of them were used to one handed operation in aircraft 
with relatively light control forces. Once the up and away training was completed, each of the pilots 
practiced the approach procedure with the hazardous failure case icing condition as would be performed 
in the actual evaluation. A certain amount of time was also spent having the pilots practice the missed 
approach procedure under iced conditions. This was the most difficult task to perform because of high 
pitch control forces and the requirement to null the asymmetric thrust with high rudder forces when an 
engine was failed. Pilots had to practice this maneuver several times before they could consistently 
accomplish it in a proficient manner and not lose control. Evaluation pilots were continually reminded to 
fly according to the SOP, which was intentionally designed to place the aircraft near the edges of the 
flight envelope at all times. This protocol forced considerable attention to precise aircraft control 
throughout the approach task, which was essential for making a good utility assessment of ICEPro. The 
evaluation profile approach task for both groups required considerable mental concentration, physical 
strength and very aggressive control inputs to prevent or recover from upset conditions. All pilots, 
regardless of size or gender, were able to effectively perform the approach task where pitch control forces 
reached 90 to 100 lbf at times. In general, most pilots only required one practice approach before they 
were ready to begin the evaluation phase. In addition to the flight training, pilots were briefed on and 
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practiced completing the NASA TLX workload form. Following this training, pilots were critiqued by the 
test team, and then given a short break before they began their evaluation runs. 

4.4 Flight Evaluations 

The evaluation phase required that a total of three satisfactory approach tasks be performed by each 
pilot. The test conductor sat behind the simulator cab and acted as ATC and was able to view the evaluation 
pilot through a small opening in the rear curtain. During each evaluation run, the pilots were completely 
enclosed in the simulator cab, and no coaching or extraneous discussions were permitted. As shown in 
Figure 4, the approach task consisted of “radar vectors” to intercept a precision approach procedure. The 
pilot would execute the approach according the SOP, and “break out” of the clouds at 400 ft above ground 
(AGL), and continue the descent to the runway on a visual glide path. At approximately 100 ft AGL, “ATC” 
would instruct a missed approach procedure, which the pilot would acknowledge and then initiate. As soon 
as both power levers were fully advanced, the test conductor would fail an engine, and the pilot would 
continue the procedure while raising the wing flaps from full down to 10°. The test would be terminated 
when the evaluation pilot was stabilized on the missed approach heading. Once complete, the evaluation 
pilot would fill out an electronic TLX form, which automatically computed the numerical workload score, 
and receive a short debriefing from the test team. The pilot would then re-enter the simulator for the next 
evaluation run. The entire evaluation process for each pilot including the pre-test training exercises and three 
evaluation runs took approximately 3 hr, which seemed to be the limit before fatigue became a factor 
affecting pilot performance. After testing was completed for each pilot, they were asked to fill out a post-test 
survey questionnaire, included in Appendix A. This questionnaire consisted of four sections using a Likert 
(Ref. 1 8) style format: Part 1 provided pilot demographic and experience information; Part 11 asked about 
situational awareness; Part 111 asked for the experimental group’s assessment of ICEPro integration; and 
Part IV asked general questions about workload factors. 

4.5 Data Acquisition and Handling 

Pilot flight performance data parameters, which were available through a flight data file that was 
resident in D-Six, were sampled at 10 Hz. MATLAB (The MathWorks, Inc.) routines were used to collect 
and reduce these data for further analysis. For the final approach segment of the task, the Theil coefficient 
was used to evaluate how closely the pilot’s flight path and airspeed agreed with that of a perfectly flown 
approach. A Theil coefficient was calculated for measured lateral and vertical deviations from the 
centerline of the approach path, and the airspeed error from the required 75 kn indicated airspeed (K1AS) 
approach speed. MATLAB routines were also used to record the number of times the stick shaker and 
pusher fired, and number and depth of tail stall AOA exceedance events over the duration of the entire 
flight task. In addition to these performance data, other parameters were recorded during each run to 
provide a more complete picture of the pilot’s control strategy which was also useful when training and 
de-briefing the evaluation pilots. These included time histories of pressure altitude (H P ), pitch attitude (0), 
pitch rate (q), normal acceleration (n z ), elevator position (8 e ), elevator control force (F e ), and wing flap 
setting (8 f ). Pilots completed the NASA TLX immediately after each data run, and then completed the 
survey questionnaire at the termination of the evaluation session. 

The pilot performance data quantified how well the pilot flew the approach task, how many stick shaker 
and pusher events occurred, and the number of times and extent to which the tail stall AOA was exceeded 
by 5° or more. It should be noted that reaching a tail stall AOA, which was one of the measured parameters, 
did not always result in an upset as long as the pilot immediately recognized the condition and promptly 
arrested the negative pitch rate. The added 5° margin took into account pilot reaction time and aircraft 
dynamic response to the controls. Therefore, tail stall upsets were defined by the number of times the pilot 
reached or exceeded the 5° margin, and the maximum negative AOA achieved in the event. This latter 
metric was also used to define “upset risk”, since the deeper the stall, the greater the chance of an upset from 
which recovery was less probable. In addition to pilot performance data, the numerical scores of the TLX 
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workload assessments and the numerical results from the Likert -type post-test questionnaires were analyzed 
by the MWU process. The TLX analysis was of interest to understand if the added information of the 
ICEPro displays adversely affected workload. The post-test survey analysis was used to assess if there was a 
real and positive difference in hazardous aircraft state awareness, and perception of workload between the 
two groups. Additionally, a section of the survey provided non-comparative information on how well the 
experimental group felt the ICEPro was integrated with flight displays. 

The utility evaluation of ICEPro was based upon assessing real differences between the control and 
experimental groups. The MWU analysis process began by ra nk ordering the scores of each group, and 
computing a “U” value based on the rank sums. In this one directional test, the “U” for the experimental 
group had to be greater than 144 to ensure a 95 percent probability for correctly rejecting H 0 . A cumulative 
probability p was calculated from the z score and was compared to a = 0.05, the requirement for accepting 
the alternate hypothesis. If this probability was less than 0.05, the null hypothesis was rejected. A negative z 
score (number of standard deviations) of more than magnitude 1 .69 (i.e., less than -1 .69) was required to 
produce the cumulative probability which met this requirement. The parameters which were assessed 
included all quantified measures of pilot performance, TLX workload, and subjective opinions from the 
survey questionnaires. 

An assessment of recovery from tail stall upsets was also performed. In many cases, when a tail stall 
event was encountered, a series of pitch oscillations would ensue from which the pilot would have to 
recover. An analysis of repeated events was performed to determine if the experimental group could 
recover from these oscillations using ICEPro cueing sooner than the control group, and thereby be less 
prone to entering an out of control condition. A pilot coupling event (PCE) represents a hazardous 
situation, which if not arrested quickly, could progress to a loss of control. In this context, the risk of an 
unrecoverable upset was associated with the number of repeated tail stalls in a PCE scenario, and the 
depth of each stall beyond the 5° margin previously discussed. A time window analysis was therefore 
used to analyze the number of tail stalls occurring within a continuous 5, 10, 15, and 20 sec span, while 
the pilot was attempting a recovery. Search scripts in MATLAB were used to analyze the beginning and 
ending of each PCE, and total the number and severity of each tail stall within the PCE. The results were 
then reported in a tabular format for both groups, and analyzed for significance by the MWU method. An 
important outcome of this analysis was to assess the utility of ICEPro AOA cues for minimizing the 
recovery time from a PCE, and thereby reduce risk of a loss of control situation. 

5.0 Results and Discussion 

5.1 Pilot Demographics 

All evaluation pilots were instructor pilots in the ERAU professional pilot training program. Thirty 
pilot volunteers were broken down into two groups — a control group who flew baseline displays, and an 
experimental group who flew with ICEPro modified displays. The 29 ERAU instructor pilots held FAA 
commercial licenses with multi-engine and instrument ratings. The demographic data was collected in 
Part 1 of the post-test survey questionnaire, and is discussed in Section 5.6. Due to a scheduling problem, 
the control group consisted of one less pilot that the experimental group. The demographics of the groups 
were as follows: 

• Subjects and gender 

- 14 control group (12 male, 2 female) 

- 15 experimental group (12 male, 3 female) 

• Median flight hours 

- Control group Experimental group 

• Total 1350 Total 1250 

■ Multi-engine 122 Multi-engine 100 
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Icing related experience and training as related from Part I of the post-test survey questionnaire: 

• 75 percent had no prior in-flight icing experience 

• 89 percent did not feel that their prior icing related knowledge/experience would have prepared 
them to perform the test scenario (training was required in order for pilots to execute the flight 
task) 

• 72 percent agreed that the NASA videos/web based training gave more information about icing 
than they had known before (viewing NASA web based training videos was a pre-test 
requirement) 

• Mixed responses were given for icing related flight training being focused on IPS operation (this 
depended on the types and classes of aircraft in which pilots had been previously trained) 

5.2 Pilot Control Performance 

Pilot control performance assessment from the control and experimental groups are summarized in 
Figure 5. Results highlighted in green indicate that the significance or real difference criterion for the 
assessed parameter was met, and the performance of the experimental group was better than that of the 
control group. Results highlighted in blue indicate that the assessed parameter showed better performance 
by the experimental group, but the result did not meet the real difference criterion. Results highlighted in 
orange, indicate that there was no difference between the performance of the two groups, and results 
highlighted in magenta indicate that the experimental group did not perform as well as the control group. 
In general the cumulative pilot performance for the two most critical events, pusher activation and tail 
stall, met the real difference criterion. These events were considered most important from a safety of 
flight standpoint because a wing or tail stall upset had occurred, and potential for loss of control was very 
high unless the pilot promptly applied the correct sequence of recovery controls. The shaker activation 
event indicated that the control group had slightly better performance than the experimental group, but 
this finding can be misleading without considering important aspects of shaker implementation and pilot 
use of this functionality in a human factors context. This is explained in the following discussion of the 
results, which were tabulated in Figure 5. The first column of the figure identifies the parameter or event 
that was evaluated during the approach and missed approach task, the second column indicates the 
number of standard deviations (z) required in meeting the real difference criterion, the third column 
indicated the cumulative probability ( p ), and the last column provides a performance assessment between 
the experimental and control group based upon the analysis. 

5.2.1 Shaker Events — Stall Warning 

The results of the pilot performance for avoiding stick shaker events as shown in Figure 5 indicate 
that, cumulatively, the control group had a slightly better performance than the experimental group. 

Shaker firings typically occurred in banked turns if speed got too low, during positive pitch overshoots 
incited by a tail stall induced PCE, and when pilots made an overly aggressive stick pusher recovery. The 
following discussion places these results into perspective and discusses factors that may have affected 
their outcome. The control group flew with basic aircraft displays and a simple stick shaker functionality, 
which always provided a 7 percent stall speed margin based upon a clean wing; i.e., no-ice. Therefore, 
when the shaker fired during the approach task in the icing condition, the actual margin to stall was less 
than 7 percent. On the other hand, the experimental group flew with ICEPro generated displays, which 
provided real time AOA cues and shaker functionality that maintained a 7 percent stall speed margin for 
the icing condition. Experimental group pilots could avoid shaker events if they remained below the upper 
alpha limit as shown in Figure 2. The control group had no AOA information on their basic flight 
displays. In order to avoid shaker activation, they had to closely maintain the SOP required airspeeds 
throughout the entire task. 
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Pilot control 
performance by run 
number and 
cumulative 

z score: 

z< -1.69 for 95% 
confidence in H 0 
rejection 

Probability: 

p < 0.05 required to satisfy a 
(alternate hypothesis) 

Group performance: Exp. versus 
control 

Positive real difference (PRD), 
Better (B), Worse (W), Same (S) 

Shaker events 

Run 1 

1.0474 

0.8575 

W 

Run 2 

-0.2837 

0.3799 

B 

Run 3 

-0.5892 

0.2705 

B 

Cumulative runs 

0.0873 

0.5434 

S 

Pusher events 

Run 1 

-0.0655 

0.4652 

S 

Run 2 

-1.2875 

0.0952 

B 

Run 3 

-2.1822 

0.0137 

PRD 

Cumulative runs 

-1.7457 

0.0385 

PRD 

Stall Messages 

No stall messages appeared for the experimental group 

Tail stall events 

Run 1 

-1.2657 

0.0989 

B 

Run 2 

-1.8330 

0.0318 

PRD 

Run 3 

-1.7894 

0.0350 

PRD 

Cumulative runs 

-2.1822 

0.0137 

PRD 

Legend 


Meets 95% confidence for rejecting H 0 


Experimental group better over control group but < 95% confidence for rejecting H 0 


No difference between experimental and control group 


Experimental group performance worse than control group 


Figure 5. — Summary of pilot control performance 


Notably, the control group was technically at greater risk when the shaker fired because their stall 
speed margin was less than 7 percent. A study of the frequency of shaker events is helpful in order to 
better understand the results of Figure 5. Table 1 provides the number of recorded shaker events per run 
for each group of pilots. 


TABLE 1.— NORMALIZED SHAKER EVENTS 


Control group 
(14 pilots) 

Experimental group 
(15 pilots) 

Run 

Shaker events 

Nonnalized 

Run 

Shaker events 

Nonnalized 

1 

87 

6.21 

1 

113 

7.53 

2 

92 

6.57 

2 

86 

5.73 

3 

87 

6.21 

3 

72 

4.80 

Cum 

266 

6.33 

Cum 

271 

6.02 


During run one, 14 control group pilots experienced 87 shaker events, whereas 15 experimental group 
pilots experienced 113 shaker events. On average, the control group experienced 6.21 events per pilot, 
and the experimental group experienced 7.53 events per pilot. The corresponding MWU analysis for run 
one in Figure 5 clearly shows better (but not significant) performance from the control group. During the 
subsequent runs, however, the experimental group’s performance got better, and this is reflected in the 
analysis for runs two and three in Figure 5. If all three runs are considered cumulatively, the normalized 
ratio between the two groups is slightly better for the experimental group, but the cumulative results of 
the statistical analysis shown in Figure 5 slightly favors the performance of the control group. Regardless, 
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the differences between control and experimental groups as shown in Figure 5 and Table 1 are very small. 
What is important to observe here is that the experimental group did a much better job of avoiding shaker 
events during subsequent runs, while the control group’s performance remained nearly constant. This was 
likely due to increased proficiency in using ICEPro AOA cues, which was not fully realized during pre- 
test training. Unfortunately, practical limits on pre-test training were necessary in order to conduct all 
required evaluations within time and program cost constraints. 

5.2.2 Pusher Events — Wing Stall Upsets 

Table 2 summarizes the number of pusher events per run for each group. Pusher events were 
generally precipitated by the same factors that incited shaker events and occurred when the pilot could not 
arrest an angle of attack increase, which resulted in a stall upset. This was particularly evident when the 
missed approach procedure was executed. Here, pilots immediately advanced the throttles to full power, 
raised the wing flaps to 1 0° and the nose to the takeoff attitude in one coordinated motion. As soon the 
power levers reached the full power stop and the evaluation pilot reached for the wing flap lever, the left 
engine was failed by the test conductor, and the pilot then had to null the yaw and adjust pitch so as to 
maintain a 76 KIAS climb speed. This was a very dynamic maneuver, and in many cases it precipitated a 
PCE. As in the case of the shaker events, the experimental group of pilots improved with each evaluation 
run, whereas the control group actually got worse. The raw data in Table 2 supports the statistical 
assessment that the ICEPro displays resulted in better pilot control performance and lower risk of an upset 
due to either wing or tail stall. 


TABLE 2.— NORMALIZED PUSHER EVENTS 


Control group 
(14 pilots) 

Experimental group 
(15 pilots) 

Run 

Pusher events 

Nonnalized 

Run 

Pusher events 

Nonnalized 

1 

12 

0.86 

1 

13 

0.87 

2 

16 

1.14 

2 

10 

.67 

3 

19 

1.16 

3 

6 

.40 

Cum 

47 

1.05 

Cum 

29 

.64 


5.2.3 Stall Messages 

A “STALL” message was incorporated into the PFD in the event the pilot attempted to override the 
pusher and thereby remain in a stall upset condition. This was a functionality associated with ICEPro 
displays, and appeared to be unnecessary as all the experimental pilots promptly applied recovery controls 
to reduce the AOA below the upper bar when the pusher fired. No stall messages appeared during any of 
the runs flown by the experimental group. 

5.2.4 Tail-Stall Upsets and Recoveries 

Tail stall upsets resulted when pilots flew too fast, exceeding the speeds specified at various segments 
of the flight task, or when they allowed the angle of attack to get too low. This condition was greatly 
aggravated with increasing wing flap deflections, and was most sensitive during the final approach and 
missed approach procedure when the flaps were 30° down. As described previously, a tail stall was 
defined as reaching a negative angle of attack that was 5° below that where the tail had technically 
reached a stall condition. This margin was determined experimentally, as it appeared that if a pilot 
immediately applied recovery controls before reaching this negative angle of attack, recovery would be 
prompt and the chance of a PCE minimized. When a tail stall occurs in an aircraft equipped with 
mechanical controls, such as the one simulated in this test, the pilot feels an immediate increase in pull 
force and notes a rapid build-up in negative pitch rate. In order to arrest the condition, the pilot must 
quickly exert very high pull forces (on the order of 90 to 1 00 lbf) to reverse the downward motion of the 
nose. If successful, the nose starts back up; but, as it does, pitch forces immediately lighten and a positive 
pitch overshoot can easily occur. This can precipitate a PCE, which can develop into a loss of control 
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situation should a negative angle of attack be reached, which the pilot is not able to physically overcome. 
Pre-test pilot training acquainted both control and experimental group pilots with these characteristics, 
and the basic methods for recovery. The control group, of course, did not have the AOA displays of the 
experimental group and therefore did not know their margins to either a wing or tail stall upset. They 
practiced tail stall upset recoveries by keeping the pitch divergence to a minimum. The experimental 
group, on the other hand, was taught to use the ICEPro displays to facilitate recovery by adjusting pitch 
during a PCE to remain within the safe envelope defined by the AOA bars. When pitch rate was negative, 
the pilot would attempt to pull the nose up but remain below the high AOA bar. If the pitch rate was 
positive, the pilot would push the nose down, but remain above the low AOA bar. If done properly, this 
would result is a very expeditious recovery — normally within one or two cycles. Of note, this was also the 
reason why there were fewer pusher events as the experimental group tended to be better at avoiding stall 
angle of attack than the control group. The learning bias evident in the shaker and pusher data seem to 
indicate that had more time been available for the experimental group to practice with the ICEPro 
displays, their performance during these events would have been better. Runs 2, 3, and cumulative of the 
tail stall event in Figure 5 clearly show positive real difference in the performance of the experimental 
group and provide better than 95 percent confidence for correctly rejecting H 0 . 

5.3 Pilot Precision Approach Performance 

The final approach segment required the pilots to intercept and accurately fly a precision approach to 
the minimum descent altitude of 200 ft. As shown in Figure 2 the flight displays for both groups did not 
provide computed steering or a flight director format. Therefore, the pilots had to fly what is termed “raw 
data” and null localizer and glide slope errors by controlling heading and pitch attitude. As described 
earlier, this task was also performed in IMC until reaching the imposed ceiling of 400 ft above ground 
level. When the pilots “broke out” of the cloud at this altitude, they continued their descent to 1 00 ft and 
were directed to execute a missed approach. The difficulty in flying the approach task was largely due to 
control aspects discussed earlier. Some pilots were very smooth and precise throughout the task, and were 
able to execute it without too much difficulty. This was because they did not allow the attitude of the 
aircraft to vary much beyond that required to maintain a given flight condition. Generally, these were 
pilots who had very efficient instrument scans, and who made very small but immediate corrections when 
required. Pilots whose instrument scans were less efficient and allowed large pitch or roll variations to 
occur were more prone to get into a pitch axis PCE. This event greatly affected glide slope, course, and 
airspeed errors, and was reflected in the Theil coefficient values for each of those parameters. Figure 6 
provides the results of the MWU analysis of the precision approach errors between the control and 
experimental groups. It should be noted, that both groups of pilots generally flew the approach procedure 
within ATP standards (Ref. 19) as required in the SOP. The Theil coefficient values, which were used in 
the MWU analysis, were basically a means of quantifying the pilot’s performance with respect to a 
perfectly flown approach. 

As shown in Figure 6, the control group had better cumulative performance in minimizing localizer 
and airspeed errors. The experimental group on the other hand had better cumulative performance in 
minimizing glide slope error, but did not meet the significance criteria required to infer that there was a 
real difference between the two groups. It would appear, after considering the results from Figure 5 and 
Figure 6, that pilots who had the advantage of ICEPro displays tended to prioritize aircraft control over 
the preciseness at which they flew the localizer and the required airspeed. This is perhaps why the control 
group, who flew more precise airspeed, had significantly more tail stall upset events than the 
experimental group and was considered to be at greater risk of entering an out of control situation. 
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Pilot precision approach 
performance by run 
number and cumulative 

z score: 

z < - 1 .69 for 95% confidence 
in H 0 rejection 

Probability: 

p < 0.05 required to 
satisfy a (alternate 
hypothesis) 

Group Performance: Exp. 
versus Control 

Positive real difference (PRD), 
Better (B), Worse (W), Same (S) 

Localizer 

Run 1 

0.6328 

0.7436 

W 

Run 2 

0.2619 

0.6116 

W 

Run 3 

0.0436 

0.5260 

W 

Cumulative 

0.1528 

0.5692 

W 

Glide slope 

Run 1 

-1.4402 

0.0718 

B 

Run 2 

0.5674 

0.7221 

W 

Run 3 

-1.4621 

0.0689 

B 

Cumulative 

-0.4364 

0.3233 

B 

Airspeed control 

Run 1 

1.8330 

0.9681 

W 

Run 2 

1.9640 

0.9764 

W 

Run 3 

1.4402 

0.9281 

w 

Cumulative 

1.9421 

0.9752 

w 

Legend 


Meets 95% confidence for rejecting H 0 


Experimental group better over control group but < 95% confidence for rejecting H 0 


No difference between experimental and control group 


Experimental group performance worse than control group 


Figure 6. — Pilot precision approach performance. 


5.4 Exposure to Risk of an Out of Control Event 

A number of tail stalls, which were encountered in the course of the approach procedure, resulted in 
PCEs before the pilot was able to stop the oscillations and recover to a stable and normal flight condition. 
This event was described earlier in Section 4.5, and under “Tail-stall upsets and recoveries,” Section 5.2. 

If a quick recovery from a tail stall event (TSE) could not be facilitated by the pilot, there was a risk 
that repeated oscillations result in a PCE which would deteriorate into an out of control situation. 
Therefore, the numbers of repeated tails-stalls during PCEs were summed for each group during the final 
approach segment in four time duration windows of 5, 10, 15 and 20 sec. The methodology began by 
identifying an initial tail-stall event (TSE), which met the criterion (negative tail-stall AOA plus 5°). If 
another TSE occurred within 5 sec it would be counted with the first tail stall and considered a PCE. If in 
this event, TSEs continued to occur within 5 sec of the one previous, they would be added to the PCE 
until the duration of time between a repeat TSE was greater than 5 sec. This methodology was then 
repeated for time intervals of 10, 15, and 20 sec. The exposure to risk of reaching an out of control 
situation was therefore greater during a PCE based upon the number of repeat TSEs in any of the four 
time frames, the amount of time it took the pilot to recover, and the maximum negative pitch attitude 
reached during the event. The results of this analysis are provided in Table 3. 
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TABLE 3.— EXPOSURE TO RISK BASED UPON PILOT COUPLING EVENTS 


Group 

Time interval 
between repeat TSEs 
(sec) 

Total 
number 
of PCEs 

Mean duration of PCEs 
in the time interval 
(sec) 

Mean max 
negative 
pitch 

Control 

5 

16 

15.16 

-12.23 

Experimental 

4 

8.66 

-8.8 

Control 

10 

69 

18.54 

-12.06 

Experimental 

29 

17.98 

-11.12 

Control 

15 

84 

23.08 

-12.32 

Experimental 

36 

23.22 

-11.37 

Control 

20 

90 

27.64 

-12.44 

Experimental 

41 

25.85 

-11.23 


The most significant events occurred during the 5 sec interval. By using their ICEPro driven displays, 
the experimental group had 25 percent fewer PCEs, their events were approximately half as long in 
duration, and the maximum nose down attitude during the event was about 3.5° less than that of the 
control group. In effect, the experimental group could arrest a PCE quicker, with fewer oscillations than 
the control group. Most important however, was the maximum depth of the tail stall, which was measured 
by the maximum negative pitch attitude achieved during the recovery attempt. The more negative the 
attitude, the higher the negative column forces, and the more difficult it was for the pilot to arrest the 
negative pitching moment. Though not shown here in the data, some pilots had to use in excess of 90 to 
1 00 lbf to arrest the negative pitching moment, which was more likely to cause a positive pitch overshoot, 
and sometimes result in pusher activation. As previously mentioned, the test profile was intentionally 
designed to place pilots on the edge of a TSE during the approach, and as a result, most pilots in both 
groups experienced them, especially when momentarily distracted from their instrument cross-check. 
However, when a TSE occurred and especially when it deteriorated into a PCE, the experimental group 
was able to effectively utilize their AOA cues to damp the oscillations more quickly than the control 
group. They accomplished this recovery while minimizing their time at an unsafe negative AOA, and 
thereby were at less risk of reaching an out of control situation. 

5.5 Workload 

After completing each evaluation run, evaluation pilots were asked to complete a TLX workload 
assessment for the entire task. The flight profile was intentionally designed to place the each group of pilots 
very near the edges of the safe flight envelope especially during the final approach and missed approach 
segments. Further, the tasks were flown in IMC, and both the cognitive and physical aspects of workload 
were quite high for both groups, especially when recoveries from TSEs or PCEs were required. Because the 
experimental group had to integrate more information when performing the flight task than the control 
group, there was a concern that their workload could be adversely affected because the ICEPro displays 
provided much more information than the basic displays, and had to be integrated into the pilot’s scan along 
with the instrument flying task. The control group did not have the additional information and thus had less 
cognitive workload. An analysis was conducted on each of the three runs for both groups, and the results 
shown in Figure 7 indicate that overall, the experimental group reported less total workload. Run 2 for 
example, taken by itself did meet the significance criteria and indicated that ICEPro displays made a real 
difference in reduced workload. However, on average for all runs, the results did not meet the significance 
criteria. Observation of both groups of pilots indicated that as they flew each successive profile, they 
became more comfortable with the task, and tended to report lower workload. In spite of the lower 
workload, it was also apparent that, by the second run, fatigue began setting in; this affected pilot 
concentration and control performance. The entire evaluation was conducted over an approximate 3 hr 
period with strategically inserted rest breaks, but it was apparent that the performance of the pilots from both 
groups seemed to peak by the second evaluation and decrease on the third run. The results of the post-test 
survey questions, which are discussed in the next section, provide further insight into these findings. 
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Pilot workload by 
run number and cumulative 

Z score: 

z<-1.69 for 95% 
confidence in H 0 rejection 

Probability: 

p < 0.05 to satisfy a 
(alternate hypothesis) 

Group Performance: 
Exp. versus Control 

Positive real difference 
(PRD), Better (B), 
Worse (W), Same (S) 

TLX Workload 

Run 1 

-0.5237 

0.2926 

B 

Run 2 

-1.8985 

0.0274 

PRD 

Run 3 

-1.4621 

0.0689 

B 

Average all runs 

-1.0911 

0.1328 

B 

Legend 


Meets 95% confidence for rejecting H (l 


Experimental group better over control group but < 95% confidence for rejecting H 0 


No difference between experimental and control group 


Control group performance better than experimental group 


Figure 7. — Workload assessment. 


5.6 Post-Test Survey Results 

The questions in all four parts of the survey solicited pilot opinions to help gain insight of the two 
pilot groups with respect to their icing experience, perception of situation awareness, how well ICEPro 
was implemented, and workload issues associated with the flight task. Descriptive analysis was used for 
responses to Part 1 — Demographics and Part 111 — ICEPro Implementation. Parts II — Situation Awareness, 
and Part IV — Workload, were analyzed via the MWU methodology for real differences. The survey 
questions solicited either agreement, or frequency of occurrence. A Likert five-answer format was used 
with numbers 1 to 5 assigned to the answers from left to right. For the MWU analysis of Parts II and IV, 
the “direction” of the numbered response for each question, i.e., 1 to 5, or 5 to 1, was defined to indicate 
better performance. The survey questionnaire is provided in Appendix A for the reader’s reference, and 
histograms of the responses to questions in Parts I, II and IV are provided in Appendix B. Descriptive- 
only assessments of experimental group pilot opinions are provided for Part III. 

5.6.1 Part I — Demographics 

The two pilot groups were intentionally selected from a relatively homogeneous sample. All pilots 
were essentially General Aviation (GA) instructor pilots who shared common experience with training, 
ratings, and types of aircraft flown. All pilots except one were staff instructors at ERAU and subscribed to 
the same operational and flight standards as defined by the University’s professional pilot training 
program. There was only one outlier in the control group, and that individual’s flight hours were not 
included in the standard deviation calculations as this individual’s flight time skewed the data 
considerably. With the elimination of that pilot’s flight hours, the standard deviation of the control group 
was 1054 hr, and the standard deviation of the experimental group was 698 hr. As shown in Figure B.l 
nearly all pilots had minimal experience with aircraft icing or icing related training before participating in 
this test. 

5.6.2 Part II — Situation Awareness 

Figure 8 and Appendices A and B summarize the situational awareness responses to the questions. 
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Pilot post-test survey 
questionnaire results 

z score: 

z< -1.69 for 95% 
confidence in H 0 
rejection 

Probability: 

p < 0.05 required to 
satisfy a (alternate 
hypothesis) 

Group Performance: 
Exp. versus Control 

Positive real difference 
(PRD), Better (B), 
Worse (W), Same (S) 

Part II — Situational Awareness 

1 . Flight displays were adequate to determine if 
airframe icing was having an effect on 
aircraft characteristics. 

-3.0987 

0.00008 

PRD 

2. Knew minimum safe speed for a given wing 
flap setting within 5 kn. 

-2.2242 

0.0072 

PRD 

3. Knew how to adjust pitch attitude to avoid a 
wing stall or a tail stall upset. 

-2.5750 

0.0047 

PRD 

4. Knew wing flap settings for safe rate of 
climb in event of an engine failure. 

-2. 5968 

0.0044 

PRD 

5. Had to rely on aircraft control response to 
determine icing effects on pitch, yaw, and 
roll. 

-4.4298 

0.0000 

PRD 

6. Confident that the final approach airspeed 
would prevent stall. 

-0.3273 

0.3634 

B 

7. Able to avoid aircraft handling problems or 
tail stall upsets when selecting wing flaps 
down. 

1.1129 

0.8717 

W 

8. Colored bands on airspeed tape useful to 
safely fly the aircraft during the entire flight. 

-2.7714 

0.0026 

PRD 

9. Relied on stick shaker to prevent 
inadvertently stalling. 

-0.1746 

0.4221 

B 

10. Always knew when approaching a wing or 
tail stall condition. 

-1.8985 

0.0274 

PRD 

Legend 


Meets 95% confidence for rejecting H 0 


Experimental group better over control group but < 95% confidence to reject H 0 


No difference between experimental and control group 


Experimental group performance worse than control group 


Figure 8. — Situational awareness post-test questionnaire results. 


aircraft control response. From the responses to question 8 the experimental group of pilots felt the 
colored bands on the airspeed tape were useful for safely flying the aircraft during the entire flight, while 
the control group did not. The histogram of the responses, Figure B.3, clearly reflects this difference, but 
the result was somewhat confusing since both groups had the same color bands on their airspeed tape 
(Figure 2), and a real difference was not expected. One explanation however, may have been that the 
ICEPro speed carets, when overlaid on the baseline PFD display speed tape, provided better low and high 
speed awareness in relation to the non-iced aircraft condition. Another possibility is that the experimental 
group misinterpreted the question and thought the “color bands” referred to the ICEPro speed carets. In 
any case, the wording of the question could have been better to ensure that no misinterpretation of the 
nomenclature was made. Question 10 indicated that the experimental group felt they had significantly 
better awareness when approaching a flight condition that could lead to a wing or tail stall than the control 
group. Many of the experimental pilots verbally commented during their test runs that the real time 
computed AOA brackets were most useful in that respect. The responses to questions 6 and 9 
respectively, indicated that the experimental group was more confident that their final approach airspeed 
would avoid wing stall, and that they did not have to rely as much on the shaker to avoid stalling. These 
results, although better for the experimental group, did not meet the significance criteria. This better but 
not significantly better result may have been due to the fact that the experimental group knew that they 
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could safely fly to just below the AOA upper bracket and avoid the shaker firing. If the brackets started 
flashing coincident with the aircraft reference symbol and the shaker did fire, they could immediately 
reduce AOA and avoid stall. The combination of AOA and shaker gave the experimental group good low 
speed and incipient stall awareness. The control group on the other hand, had to rely on basic (clean 
aircraft) shaker settings and the likewise basic (clean aircraft) airspeed bands on their PFD to avoid 
stalling. 

Fortunately for the control group, the failure case icing configuration in the simulation model resulted 
in a very small stall speed increase and concurrent stall AOA decrease. As a result, if the shaker fired for 
the control group, or if they flew into the amber stall warning band on the airspeed indicator, they were 
generally able to react with a decreased pitch input in time to avoid stall. Flad a more severe ice shape 
been modeled and the stall margin been less, or even negative, the control group responses may have been 
different as they could have stalled before the shaker fired. The responses to question 7 indicated that the 
experimental group felt less able to avoid handling problems or tail stall upsets while extending the wing 
flaps than the control group. Referring to Figure B.3, it is apparent that the greatest number of responses 
from the control group agreed that they could manage their control inputs to avoid handling problems, 
while there were an equivalent number of disagreement responses from the experimental group. This 
result was somewhat puzzling because it was apparent from the data in Figure 5 that quantitatively, the 
experimental group was significantly better than the control group in avoiding tail stalls. One possible 
explanation is that the experimental group was provided with displays that clearly showed the effects of 
wing flap extension on the safe AOA envelope, but were required by the test protocol to ignore those 
indications and extend the wing flaps regardless. This could well have led to their response choice 
because they expected handling problems as wing flaps were extended, and knew that they were 
unavoidable. The question may have been poorly worded in that respect. 

5.6.3 Part III — ICEPro Integration 

The experimental group was asked fifteen questions regarding the integration of ICEPro displays. 
Since these questions only affected that group, a descriptive statistical approach was taken in the analysis 
of the data. The primary goal of these survey questions was to understand if the implementation of real 
time state assessment into flyable pilot cues did in fact provide utility of ICEPro for the pilot. The key 
findings that follow list the percentage of responses on either side of a “neutral” opinion. 

1 . 40 percent agreed, 40 percent disagreed, that messages were not useful and cluttered the display 

2. 53 percent agreed, 33 percent disagreed that combining visual, aural and tactile cues was important 

3. 60 percent agreed, 40 percent disagreed that bars were useful to fly safely 

4. 70 percent agreed, 20 percent disagreed that it was difficult to understand relation between AOA 

bars and Hi/Lo airspeed carets 

5. 40 percent agreed, 40 percent disagreed that synoptic page was more useful than messages for state 
assessment 

6. 47 percent frequently or always, 40 percent never or infrequently favored using airspeed carets over 
AOA bars for safe flight 

7. 40 percent agreed, 47 percent disagreed that decluttering did not affect ability to recover from stall 

8. 40 percent agreed, 47 percent disagreed that the “red” CLM and FLP LIM needed less flap setting 

for engine out performance. 

9. 47 percent agreed, 40 percent disagreed that “control buzzing” made aircraft control difficult 

10. 60 percent agreed, 30 percent disagreed that baseline airspeed tapes and carets were confusing 

1 1 . 60 percent frequently or always, 40 percent never or infrequently flew in middle of AOA bars 

12. 47 percent frequently or always, 30 percent never or infrequently felt that AOA bars immediately 
captured attention when flashing. 

13. 40 percent frequently or always, 47 percent never or infrequently felt airspeed cues were helpful for 
maneuvering/approach. 
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14. 60 percent frequently or always, 40 percent never or infrequently felt low a cue helped avoid tail 
stall upset 

15. 67 percent agreed, 20 percent disagreed that they immediately noticed when CLM and FLAP L1M 
messages were on at the same time 

Summarizing these key findings, the experimental group felt that the AOA brackets were very useful 
in all phases of flight, and were effectively able to use the brackets to keep the aircraft in a safe flight 
envelope. But many had difficulty understanding the relationship between angle of attack and airspeed. 
None of the participating pilots had experience with flying angle of attack prior to this test, and it is felt 
that this response was largely the result of this lack of experience. More pilots felt that presenting both the 
baseline airspeed along with the ICEPro generated airspeed carets was confusing. The original design 
intent of the ICEPro displays was to present the computed real time speeds as “advisory” information. 

This way, the pilot would always have a reference to the baseline aircraft, which would prove useful if a 
system anomaly was encountered. There was a concern for the manner in which this would be done to 
avoid confusion and the survey seemed to bear this concern out. Along those same lines, pilots were 
essentially ambivalent as to the usefulness of the ICEPro speed carets when maneuvering and flying the 
approach task as shown in Figure 6. A majority of pilots felt that the integration of visual, aural, and 
tactile cues was important, but providing redundant messages on two displays was not necessary. Lastly, 
one major concern was the assessment of the multi-axis control excitation or “buzzing”, which was 
commanded by the ICEPro logic for aircraft state assessment. Some pilots were bothered by the buzzing, 
and some were not. It appeared that because it was expected, pilots accommodated to this as they would if 
having to fly through turbulence. In any case, there did not seem to be a strong opinion one way or the 
other regarding control buzzing. 

5.6.4 Part IV— Workload 

Workload assessments via the NASA TLX format were gathered after each run for each pilot. Post- 
test questionnaires were solicited in an attempt to understand some of the important factors that 
influenced the scores that pilots gave to workload assessment. The survey answers from the control and 
experimental was assessed via the MWU methodology. Figure 9 provides the results from the survey 
questionnaire. 

Pilot workload was not measurably affected by ICEPro cueing. The data indicated the following: 

1. Controlling vertical speed on glide slope was more difficult for experimental group (60 versus 
43 percent). It appears that the control group prioritized safe flight over strict airspeed control. 

2. Flight control inputs were more physically demanding for the control group (73 versus 60 
percent), possibly due to the fact that the control group was not aware of the magnitude of angle 
of attack divergence. 

3. The control group reported that it took more concentration and effort for flying MAP safely (87 
versus 80 percent). The results however did not meet the significance criterion, but did show that 
the experimental group did better. 

4. Staying on glide slope without pitch upset was more difficult for control group (79 versus 

67 percent). This was likely because the control group made corrections without knowledge of 
AOA limits. On the other hand, the experimental group had a real time display of AOA limits and 
could remain within those limits while correcting to the glide path. By “respecting” these limits, 
the experimental group tended to give greater priority to remaining in a safe flight conditions and 
were less aggressive in maintaining a precise vertical flight path. It should be noted however, that 
both groups flew the approaches within ATP standards as previously discussed. 

5. Maintaining glide slope detracted from localizer control. This affected the experimental group 
more than the control group (64 versus 47 percent). This is thought to be a result of AOA cueing, 
which captured the pilot’s attention. 
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Pilot post-test survey 
questionnaire results 

z score: 

zc-1.69 for 95% 
confidence in H 0 
rejection 

Probability: 

p < 0.05 required to 
satisfy a (alternate 
hypothesis) 

Exp. versus Control 

Positive real difference 
(PRD), Better (B), 
Worse (W), Same (S) 

Part IV — Workload 

1 . It was difficult to control vertical speed 
when on glide slope. 

0.7856 

0.7902 

W 

2. Operating the flight controls to avoid an 
upset condition when on final approach 
was a physically demanding effort. 

-0.5455 

0.2852 

B 

3. I felt it took a considerable amount of 
concentration and effort to safely execute 
the missed approach procedure. 

-1.2002 

0.1108 

B 

4. Keeping the aircraft on the glide slope 
without experiencing a pitch upset was a 
very demanding task. 

-0.8074 

0.2034 

B 

5. On final approach I spent so much time 
trying to fly the glide slope that I was 
unable to maintain good localizer course 
control. 

0.5674 

0.7221 

W 

Legend 


Meets 95% confidence for rejecting H 0 


Experimental group better over control group but < 95% confidence for rejecting H 0 


No difference between experimental and control group 


Control group performance better than experimental group 


Figure 9. — Pilot workload survey results. 


In summary, the workload was not measurably affected by the addition of ICEPro system. Pilots 
seemed to accommodate to those displays rather quickly during their training, and appeared to use them 
quite effectively to perform the flight task. TLX data showed clearly that the flight task was very 
demanding for both groups of pilots, and the physical difficulty of it far outweighed other workload 
dimensions. 

6.0 Conclusions and Recommendations 

A pilot-in-the-loop simulator evaluation was conducted to assess the utility of ICEPro for mitigating a 
hazardous icing encounter. Most of the results of this testing were statistically quantified for real 
differences between a control and experimental group by the Mann-Whitney “U” test, which is a non- 
parametric, independent measures analysis of data used to define pilot control performance, pilot opinion, 
and workload. Other data, which was not compared between two groups, was analyzed descriptively. The 
conclusions from this test indicate that the experimental group of pilots, who flew with ICEPro displays, 
demonstrated significantly better control performance in dealing with aircraft handling problems under 
icing conditions and had better awareness of a hazardous aircraft state. Pilot workload was not affected by 
the additional display information provided by ICEPro, but pilot opinions of the implementation of that 
system indicate that improvements could be made. Based on the findings of this test, future testing with 
ICEPro should include the effects of environmental uncertainties, such as in-flight turbulence and wind 
gusts, on the quality-of-state assessment. Additionally, the flight displays in this test were not optimized 
from a human factors standpoint, nor were they coupled in any way with flight guidance. Based upon the 
opinions extracted from the pilot survey, pilot control performance and state awareness might be 
improved by addressing those issues in future testing. 
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Appendix A. — Post-Test Survey Questionnaire 


Part 1. Demographics 

Name: □ Male 

Pilot Number: 

Total Flight Hours: 

Multi - Engine Hours: 

FAA Ratings: Commercial |^| Multil I CF1 IZZI 
Aircraft Flown and Hours in Type: 


1^1 Female 


CFlO 


Icing related training and experience: 

1 . In my all my prior flying experience, 1 encountered in-flight icing 

Never Rarely Sometimes Very often Always 

2. 1 felt that my prior icing related knowledge and experience would have adequately prepared me to 
perform this test scenario without the familiarization training 1 received on aircraft handling 
characteristics. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

3. The NASA videos and web based training materials provided me with information about icing that 1 
had never known about before. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

4. Before this test, my icing related flight training was mostly focused on how to operate the ice 
protection system. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

Part 11. Situation Awareness 


1 . My flight displays were adequate for me to determine when airframe icing was having an effect on 
aircraft characteristics. 

Never Rarely Sometimes Very often Always 

2. 1 felt as though 1 knew the minimum safe speed 1 could fly for a given wing flap setting within 5 knots. 

Never Rarely Sometimes Very often Always 

3. 1 knew how to adjust my pitch attitude to avoid a wing stall or a tail stall upset. 

Never Rarely Sometimes Very Often Always 

4. 1 wasn’t always sure which wing flap settings would allow a safe rate of climb in the event of an engine 
failure. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. 1 relied solely on aircraft control response to determine how icing affected pitch, yaw, or roll 
characteristics. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

6. 1 was confident that the final approach airspeed 1 chose to fly after 1 departed the final approach fix 
would prevent me from stalling. 
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Strongly Disagree Disagree 


Undecided Agree 


Strongly Agree 


7. While 1 was putting the flaps down during the approach and landing, 1 felt 1 could effectively manage 
my control inputs and airspeed to avoid aircraft handling problems or tail stall upsets. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

8. The various colored bands on the airspeed tape were useful for me to safely fly the aircraft during the 
entire flight. 

Never Infrequently Sometimes Frequently Always 


9. 1 relied upon the stick shaker to prevent me from inadvertently stalling. 

Never Rarely Sometimes Very Often Always 

10. 1 could always tell when 1 was approaching a wing or tail stall condition. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 


Part HI. Implementation of Envelope Protection Cueing (Experimental ICEPro Group Only) 

1. 1 felt that it was important to combine the visual cues from ICEPro with aural alerts, and tactile 
feedback from the stick shaker. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

2. The ROL DGRD, PTCH DGRD, or YAW DGRD messages on the PFD did not provide useful 
information, and only cluttered the display. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

3. When flying at slow speeds, I could easily use the AOA bars as a good pitch control reference for 
safely flying the airplane. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

4. 1 had difficulty understanding the relationship between the AOA bars and the high and low speed carets 
on the airspeed tape. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. The color coded flight control surfaces on the flight control synoptic were more useful than the 
messages for assessing degraded control state. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

6. 1 favored using the airspeed carets rather than the AOA bars to remain within a safe operating 
condition. 

Never Infrequently Sometimes Frequently Always 

7. When 1 unintentionally got into an upset condition, the disappearance of all ICEPro messages except 
the AOA bars on the PFD did not affect my ability to recover the airplane. (Answer only if a wing stall or 
tail stall upset occurred). 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

8. When 1 extended the wing flaps and the red CLIMB L1M and FLAP L1M messages came on together, 1 
immediately knew if an engine failed 1 would have to reduce my flap setting in order to climb. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 
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9. The occasional “buzzing” of the flight controls by ICEPro made flying the aircraft very difficult. 
Strongly Disagree Disagree Undecided Agree Strongly Agree 

10. 1 tended to confuse baseline airspeed limits with the airspeed limits (carets) that were posted by 
ICEPro. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

11.1 tended to fly so as to keep my pitch attitude in the middle of the AOA bars when they were 
presented by ICEPro 

Never Infrequently Sometimes Frequently Always 

12. When the AOA bars started flashing, they immediately captured my attention. 

Never Infrequently Sometimes Frequently Always 

13.1 felt that the airspeed carets from ICEPro were useful for helping me determine safe flight speeds 
when maneuvering during approach and landing. 

Never Infrequently Sometimes Frequently Always 

14. The low AOA cue on the PFD enabled me to avoid a tail stall upset. 

Never Infrequently Sometimes Frequently Always 

15. 1 immediately noticed when the CLIMB L1M and FLAP L1M messages were on at the same time. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 


Part IV. Workload 

1. 1 found it difficult to control vertical speed when on glide slope. 

Never Infrequently Sometimes Frequently Always 

2. Operating the flight controls to avoid an upset condition when on final approach was a physically 
demanding effort. 

Never Infrequently Sometimes Frequently Always 

3. 1 felt it took a considerable amount of concentration and effort to safely execute the missed approach 
procedure. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

4. Keeping the aircraft on the glide slope without experiencing a pitch upset was a very demanding task. 

Strongly Disagree Disagree Undecided Agree Strongly Agree 

5. On final approach I spent so much time trying to fly the glide slope that I was unable to maintain good 
localizer course control. 

Never Infrequently Sometimes Frequently Always 
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Response Frequency Response Frequency 


Appendix B. — Results of Survey Questions from Parts I, II, and IV 


■ Control Group (n=14) 



12345 12345 12345 12345 

1-1 1-2 1-3 1-4 


Figure B.l. — Pilot demographics — Part I Survey questions 1 to 4. 
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■ Experimental Group (n=15) 
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*Direction of responses that indicated better situational awareness 
Figure B.2. — Situational Awareness — Part II Survey questions 1 to 5 
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Response Frequency Response Frequency 


aControl Pilot Group (n=14) 
■ Experimental Group (n=15) 



Figure B.3. — Situational Awareness — Part II Survey questions 6 to 10 


■ Control Group (n=14) 

■ Experimental Group (n=1 5) 



* Direction of responses that indicated lower workload 
Figure B.4. — Workload — Part IV Survey questions 1 to 5. 
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of potentially hazardous upset events and in lessening exposure to loss of control following an incipient upset condition. Pilot workload with 
the added ICEPro displays was not measurably affected, but pilot opinion surveys showed that real time cueing greatly improved their 
situation awareness of a hazardous aircraft state. 
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